A Heuristic Method for Chinese Segmentation

Research and development in digital library includes content creation, conversion, indexing, organization, and dissemination, where the key technological issues are how to search and display desired selections from and across large collections effectively [10]. A repository is an indexed collection of objects. Indexing is an important task for searching. The better the indexing, the better the searching result. The smallest indexing units in Chinese digital library are words, while the smallest units in a Chinese sentence are characters. However, Chinese text has no delimiter to mark word boundaries as it is in English text. In English or other languages using Roman or Greek-based orthographies, spacing often reliably indicates word boundaries. In Chinese, a number of characters are placed together without any delimiters indicating the boundaries between consecutive characters. Previous work of Chinese word segmentation can be divided into three categories: (a) statistical approach, (b) lexical rule-based approach, and (c) hybrid approach based on statistical and lexical information. In this paper, we focus on the statistical approach. In particular, we investigate the heuristic approach based on significance estimation for Chinese segmentation. Experiments are conducted to evaluate its performance. The result of the heuristic approach is compared with that of the boundary detection approach. The heuristic approach has shown improvement in segmenting the unknown words.